Incremental reprocessing and S3 object storage (bucket storage v4 - Service v1.26.0) - #611

Resolves:

• [Drift] MongoDB storage version 3 promoted to stable as storage version 4 (powersync-service #776) #605
• [Drift] New S3 storage stats add per-sync-config Prometheus metrics (powersync-service #770) #603
• [Drift] New --incremental-only compact flag added; MongoDB V3 storage compaction restructured (powersync-service #752) #570

Also related:

• Custom Checkpoint Request Rollout Docs #614

This PR documents V4 storage and the two new Beta features built on top of it:

• Incremental reprocessing
• S3 object storage

Summary

• Added a storage version 4 page, with incremental reprocessing and S3 object storage as sections of it.
• Added an availability table that shows, per feature, which source databases apply, whether it works with Sync Streams or Sync Rules, and what Cloud and self-hosted users get. The Cloud and self-hosted columns are explicitly scoped to the Beta since we're rolling things out gradually.
• Documented how to opt in on Cloud and self-hosted, and what to expect: a full reprocess in the background, one client re-sync, a new deploy event, and how to move back to version 2. Self-hosted deployments with many clients are told to scale out the API before the switch.
• Explained incremental reprocessing using the "only the streams you added or changed" model: add, remove, rename and edit semantics, the correctness-first fallback, the cases that always force a full reprocess, and the expectation that changing one large stream still takes as long as before.
• Documented the distinction between Sync Config versions and replication streams, how the Dashboard label and log identifier relate, and how to tell an incremental deployment from a full reprocess, including the replication log summary.
• Explained S3 object storage around its benefit (initial sync at scale, less load on the storage database). Cloud users get it enabled per instance on request; for self-hosted users we added a new setup section.
• Consolidated the self-hosted S3 setup with the internal PowerSync Service S3 Config & Setup doc: bucket in the same region as the Service, versioning and Object Lock off with no expiration rule, api.parameters.max_concurrent_connections: 1000 (default 200) as part of the setup, and post-deploy verification steps. The AWS-specific parts (VPC endpoint, IAM policy, ECS roles, multipart lifecycle rule) stay in the internal doc.
• Updated the compacting guidance for storage version 4: the compact job decides per bucket between merging new chunks and a full compaction, so an hourly schedule is recommended. Removed the compact --incremental-only note, since the flag is a no-op on version 4 and only skips version 1 and 2 compaction. Technical Details now links to both service repo documents, including the version 4 compaction design.
• Trimmed the storage version reference to the concept, config.storage_version, defaults, and a versions table listing 1, 2 and 4. Removed all mentions of storage version 3.
• Reworked the defragmenting guidance to distinguish targeted source-row updates from the Dashboard's Defragment action, which forces a full reprocess even when incremental reprocessing would apply.
• Applied "Sync Config" casing consistently and added it to the terminology table.

Assumptions and decisions

• Storage version 4 is in Beta in v1.26.0 and version 2 remains the default. GA means version 4 becomes the default for all instances and no opt-in is needed.
• Cloud rollout during the Beta: Free plan instances are upgraded automatically, other plans opt in with config.storage_version: 4. S3 object storage is enabled per instance on request during the Beta and for all Cloud instances at GA. Self-hosted deployments configure S3 themselves.
• Incremental reprocessing requires a MongoDB source database, Sync Streams, MongoDB bucket storage, and storage version 4. It has no separate setting. It is not supported for Sync Rules, and Postgres and other sources are planned (proposal Docs updates in preparation for the Sync Streams beta release #349).
• S3 object storage requires MongoDB bucket storage and storage version 4. It works with any source database and with Sync Rules.
• Postgres bucket storage does not support storage version 4.
• Full reprocess triggers documented: the first version 4 deployment, changes to the config block (verified: compatibility options must match), the Defragment action, and replication failures.
• Storage version 3 is not mentioned anywhere. Version 4 is the same format marked stable.
• The Sync Config version vs replication stream section assumes the Dashboard will show both, as in "Sync Streams 3 (246a)", and that logs show the stream as powersync_<instance>_1_246a.
• Plans for storage version 4 on Postgres bucket storage are not stated in the docs.

Open items

• Add a back-link from the custom write checkpoints page after Custom Checkpoint Request Rollout Docs #614 merges.

🤖 Codex assisted with tracing the PowerSync Service implementation, finding related documentation, checking technical details, and drafting the changes. It also helped generate the list of changes and decisions in this PR description. Claude Code then reviewed the updates for accuracy.